Introduction to Continuous Entropy

Classically, Shannon entropy was formalized over discrete probability distributions. However, the concept of entropy can be extended to continuous distributions through a quantity known as continuous (or differential) entropy. The most common definition for continuous entropy is seemingly straightforward; however, further analysis reveals a number of shortcomings that render it far less useful than it appears. Instead, relative entropy (or KL divergence) proves to be the key to information theory in the continuous case, as the notion of comparing entropy across probability distributions retains value. Expanding off this notion, we present several results in the field of maximum entropy and, in particular, conclude with an information-theoretic proof of the Central Limit Theorem using continuous relative entropy.